STEM Pathways Student Academic Achievement 
Results for the 2014-15 School Year 


Summary 


A long-term goal of STEM Pathways is to increase student achievement in STEM subjects, especially 
among underrepresented groups such as those with low incomes, racial/ethnic minorities, and 
females. To take an initial look at the achievement of STEM Pathways students after the first year 
of program implementation, we examined results in math, science, and reading from spring 2015 
Minnesota Comprehensive Assessments, Series III (MCA-III). Program effects on these test scores 
are unlikely at this early stage with students having limited program exposure. Program effects 
could emerge later with multiple years of program exposure. 


A quasi-experimental design was used to assess potential program effects on MCA test results of 
fourth- and fifth-graders attending Minneapolis public schools where STEM Pathways was 
implemented in 2014-15. This design included a comparison group of fourth- and fifth-graders 
attending other Minneapolis public schools. Propensity score analysis was conducted to ensure that 
the treatment and comparison groups were similar on key student characteristics that could influence 
achievement test performance. This analysis technique was used to statistically equalize the 
treatment and comparison groups on six characteristics: gender, race/ethnicity, free/reduced-price 
lunch eligibility, English Language Learner (ELL) status, Special Education status, and spring 
2014 MCA-III math scores. Before the propensity score analysis, the treatment group had a higher 
proportion of ELL and Hispanic students, and a lower proportion of black students, than the 
comparison group. These differences between the groups essentially disappeared after the propensity 
score analysis was carried out for both fourth- and fifth-graders, making it more likely that any 
difference between the treatment and comparison groups in spring 2015 achievement test scores 
would be a STEM Pathways effect. 


Analyses were then conducted to determine whether spring 2015 MCA-III scores differed in math, 
science (5" grade only), and reading between the treatment and comparison group students. 
Results indicated that scores were very similar between the two groups for each of the tests in each 
grade. There was no statistically significant advantage for the treatment group on any of the tests in 
either grade. In fourth grade, the comparison group scored slightly higher (statistically significant) 
in reading. These results were confirmed using two additional propensity score analysis methods. 
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Potential treatment-comparison group differences on the 2015 MCA tests were examined separately 
within the following demographic categories: female, male, eligible for free or reduced-price 
lunch, ineligible for free or reduced-price lunch, ELL, non-ELL, Asian, black, Hispanic, and 
white. (There were too few students to conduct the analyses within the American Indian group). 
No strong patterns of differences emerged within any of these demographic categories. Overall, 
differences in test results tended to be small and most often slightly in favor of the comparison 
group. A weak but consistent pattern of MCA score differences in favor of the comparison group 
was found among white students and students ineligible for free or reduced-price lunch (higher- 
income students). That is, although mostly small and statistically non-significant, differences in 
favor of the comparison group were consistent across grades and type of test in these two 
demographic categories. 


Introduction 


STEM Pathways aims to increase youths’ interest, attitudes, learning, and achievement in STEM 
through a deliberate, interconnected, and comprehensive system of STEM learning opportunities. 
A key research question of the STEM Pathways project is the following: Does the STEM Pathways 
model enhance the short-term and long-term outcomes of populations underrepresented in 
STEM? Underrepresented groups include those with low income, racial/ethnic minorities, and 
females. Through the results of the pre-post student survey in 2014-15, we observed changes in 
interests and attitudes of STEM Pathways fourth- and fifth-graders that may be related to their 
participation in the program. We also examined fourth- and fifth-graders’ academic achievement 
in 2014-15. This was done through examining spring 2015 Minnesota Comprehensive Assessments, 
Series II (MCA-ID), results in math, science (Sth grade only), and reading. MCA tests are the 
academic achievement measures administered to students statewide each spring, and serve as 
accountability measurement tools in Minnesota for the federal No Child Left Behind law. We 
consider MCA tests to be long-term indicators of potential effects of the program. Significant 
program effects may be unlikely in the short term, but could emerge after multiple years of 
program exposure. 


Design 


In order to identify potential program effects, we compared fourth- and fifth-graders attending 
Minneapolis public schools where STEM Pathways was implemented (treatment group) to a 
comparison group of fourth- and fifth-graders attending other schools within the Minneapolis 
school district in 2014-15. To be eligible for inclusion in this analysis, students in both groups 
needed to be continuously enrolled in the same school during the 2014-15 school year, and to be 
enrolled for at least 160 days during the school year. In addition, students in both groups needed 
to have taken the MCA-III math test in both spring 2014 and 2015. 
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Comparison group fourth- and fifth-graders were drawn from all non-STEM Pathways Minneapolis 
district schools in which more than 45 percent of the students were eligible for free or reduced- 
price lunch (excluding those schools with a special focus on STEM subjects). The free/reduced- 
price lunch criterion was set to help make the composition of the schools attended by comparison 
students more similar to those attended by the treatment group (the proportion of students 
qualifying for free or reduced-price lunch was over 45% in all six STEM Pathways schools). 


As part of the quasi-experimental design for assessing potential treatment effects, propensity 
score analysis was used. This is a statistical technique used to equalize the treatment and 
comparison groups on key characteristics (other than the treatment) thought to influence student 
performance on the outcome indicators. If successful, propensity score analysis increases the 
likelihood that any differences found between the treatment and comparison groups can be 
attributed to the treatment. 


Using this technique, treatment and comparison group students were equalized or “balanced” on 
the following six characteristics: 


=m Gender 

= Race/ethnicity 

m Free or reduced-price lunch eligibility 
=m English Language Learner (ELL) status 
=m Special Education status 

= MCA-III math scale score in 2014 


As a first step in equalizing the two groups, a propensity score was calculated for each member 
of the treatment and comparison groups based on their profile on the six characteristics. A 
propensity score has a value between 0 and | based on how closely the profile of each member 
of both the treatment and comparison groups matches the overall profile of the treatment group 
on the six characteristics. The closer a member’s profile matches the overall treatment group 
profile, the higher the propensity score. (A logistic regression is used to generate the propensity 
scores with the six characteristics as the independent variables and group membership as the 
dependent variable, treatment=1 and control=0.) 


Students in the treatment group and the comparison group with extreme propensity scores (“outliers’’) 
were excluded for purposes of the student achievement analysis. These students had profiles on 
the six characteristics very different from most of the students and would have made it more 
difficult to equalize the two groups statistically if they stayed in the analysis. In the fourth-grade 
cohort, 48 comparison group students and one treatment student were excluded. In the fifth-grade 
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cohort, five comparison group students and three treatment group students were excluded. After 
these exclusions and a few others due to missing test scores, 355 fourth-graders remained in the 
treatment group and 1,067 fourth-graders remained in the comparison group. Among fifth-graders, 
331 remained in the treatment group and 1,059 remained in the comparison group. 


For the next step, we used the Inverse Probability of Treatment Weights IPTW) method to provide 
weights to each member of the treatment and comparison groups as a way to equalize the two 
groups on the six characteristics listed above. In the IPTW weighting method, individual weights 
are assigned based on each study group member’s propensity score. The formula used to determine 
the weights is //p for treatment group members and ///-p for comparison group members, where 
p is the propensity score. The results of using this weighting method is that treatment group 
members with higher propensity scores receive lower weights while treatment group members 
with lower propensity scores receive higher weights. For comparison group members, those with 
higher propensity scores receive higher weights while those with lower propensity scores receive 
lower weights. Assigning the weights in this way serves to balance or equalize the two groups on 
the six characteristics. Once these weights were determined, weighted regression analyses were 
performed to estimate the potential effects of STEM Pathways on students’ achievement test 
performance. 


Measurement of academic outcomes 


To assess potential treatment effects on academic achievement, the treatment and comparison 
group students were compared on the spring 2015 MCA-III math, science (5" grade only), and 
reading tests. MCA scale scores were used for this analysis. In fourth grade, scale scores range 
from 409 through 499 in math and from 411 through 490 in reading. In fifth grade, scale scores 
range from 515 through 586 in math, from 517 through 591 in reading, and from 501 through 
599 in science. We also compared the two groups on the proportion of students who reached the 
proficiency level on these tests (1.e., met or exceeded standards for their grade). 


Exposure to STEM Pathways programming 


About 29 hours of STEM Pathways programming was provided to both fourth- and fifth-graders 
attending STEM Pathways schools during the 2014-15 school year. Each grade received 
programming from three STEM Pathways partners. Fourth-graders received programming from 
The Bakken (6.25 hours), STARBASE Minnesota (20 hours), and The Works (3 hours). Fifth- 
graders received programming from the Minnesota Zoo (4 hours), STARBASE Minnesota (20 
hours), and The Bell Museum (5 hours). Some students likely received fewer than 29 hours of 
programming due to absences on program days. Some of the programming occurred late in the 
school year at some schools with little chance to affect students’ performance on MCA-III tests 
(administered in April and May). For example, fourth-grade students at four STEM Pathways 
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schools attended STARBASE in April or May, and students at one school attended The Works in 
late May. Fifth-graders at all six STEM Pathways schools attended The Bell Museum in April or 
May. 


Many comparison students had a small amount of exposure to STEM Pathways programming 
during the 2014-15 school year. The four-hour Minnesota Zoo program was provided to fifth- 
graders at most schools that comparison group students attended (18 of the 21 MPS schools 
comparison group students attended). Part of The Bakken program was provided to fourth- 
graders at three schools that comparison group students attended. The Works program was 
provided at one school that fourth-grade comparison group students attended and two schools 
that fifth-grade comparison group students attended. 


Treatment and comparison group characteristics before propensity score analysis 


Figures | and 2 indicate the profiles of the treatment and comparison groups on the six characteristics 
listed above prior to the propensity score analysis. Figure 1 shows the similarities and differences 
between these two groups for fourth-graders. Note that there is very little difference in the 
proportions of boys and girls, or in the proportions of Special Education students, in the two 
groups. The differences between the treatment and comparison groups is slightly larger for the 
percent eligible for free or reduced-price lunch, but the difference is not statistically significant. 
There are statistically significant differences between the groups in ELL students, racial/ethnic 
composition, and 2014 MCA-III math scores. The treatment group has a higher percentage of 
ELL students than the comparison group. With regard to race/ethnicity, the treatment group has a 
higher percentage of Hispanic students than the comparison group and a somewhat lower 
percentage of black students. Average 2014 MCA-III math scores are slightly higher in the 
treatment group than in the comparison group (testing occurred in the spring of third grade, scores 
could range from 315 through 399). 


For fifth-graders, differences between the two groups follow a very similar pattern (Figure 2). 
There is little difference between the groups in gender and Special Education. The difference 
between groups in eligibility for free/reduced-price lunch is not statistically significant. Again, 
statistically significant differences occur between the groups in ELL students, racial/ethnic 
composition, and 2014 MCA-III math scores. The differences follow the same pattern as with 
fourth-graders. In contrast to the comparison group, the treatment group has a higher percentage 
of ELL students, a higher percentage of Hispanic students, and a lower percentage of black 
students. Average 2014 MCA-III math scores are slightly higher in the treatment group than in 
the comparison group (testing occurred in the spring of fourth grade, scores could range from 
409 through 499). 
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1. Characteristics of treatment and comparison groups before propensity score 
analysis: 4" grade 


Group 
Treatment Comparison 

Student characteristics (N=357) (N=1,117) Difference? 
Gender Male 49% 50% -1 

Female 51% 50% 1 
Race/ethnicity® American Indian 2% 6% -4 

Asian 7% 6% 1 

Black 31% 37% -6 

Hispanic 36% 26% 10 

White 24% 24% 0 
Eligible for free or reduced-price lunch 79% 75% 4 
English Language Learner‘ 40% 30% 10 
Receives Special Education services 10% 11% -1 
MCA-III Math 2014 scale score: Mean? 350 347 3 


a Difference in percentage points (treatment minus control group). 


The difference between the treatment and comparison groups on the student characteristic is statistically significant (p<.05) using the 
Pearson Chi-Square Test (2-sided). 


¢ The difference between the treatment and comparison groups on the student characteristic is statistically significant (p<.05) using 
Fisher's Exact Test (2-sided). 


9 The difference between the treatment and comparison groups in MCA-III math scale score means is statistically significant (p<.05) 
using the t-test (2-tailed). 
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2. Characteristics of treatment and comparison groups before propensity score 
analysis: 5'" grade 


Group 
Treatment Comparison 
Student characteristics (N=338) (N=1,074) Difference? 
Gender Male 50% 49% 1 
Female 50% 51% -1 
Race/ethnicity® American Indian 5% 5% 0 
Asian 10% 7% 3 
Black 33% 43% -10 
Hispanic 35% 24% 11 
White 17% 21% -4 
Eligible for free or reduced-price lunch 82% 77% 
English Language Learner‘ 40% 32% 
Receives Special Education services 10% 12% -2 
MCA-III Math 2014 scale score: Mean? 449 445 4 


a Difference in percentage points (treatment minus control group). 


» The difference between the treatment and comparison groups on the student characteristic is statistically significant (p<.05) using the 
Pearson Chi-Square Test (2-sided). 


¢ The difference between the treatment and comparison groups on the student characteristic is statistically significant (p<.05) using 
Fisher's Exact Test (2-sided). 


9 The difference between the treatment and comparison groups in MCA-III math scale score means is statistically significant (p<.05) 
using the t-test (2-tailed). 


Treatment and comparison group characteristics after propensity score analysis 


Figures 3 indicates the profiles of the fourth grade treatment and comparison groups on the six 
student characteristics after equalizing the two groups through the propensity score analysis 
described earlier. Note that the differences between the groups in ELL status, race/ethnicity, and 
2014 MCA-III math scores we observed prior to the propensity score analysis have disappeared 
or become very small. The treatment and comparison groups have been successfully equalized 
on the six student characteristics; there are no statistically significant differences between the 
two groups on these characteristics. 
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3. Characteristics of treatment and comparison groups after propensity score 
analysis: 4" grade 


Group 
Treatment Comparison 
Student characteristics (N=355) (N=1,067) Difference? 
Gender Male 50% 50% 0 
Female 50% 50% 0 
Race/ethnicity American Indian 4% 2% 2 
Asian 7% 7% 0 
Black 37% 37% 0 
Hispanic 28% 30% -2 
White 24% 24% 0 
Eligible for free or reduced-price lunch 76% 76% 
English Language Learner 32% 34% -2 
Receives Special Education services 11% 11% 0 
MCA-III Math 2014 scale score: Mean 347 348 -1 


a Difference in percentage points (treatment minus control group). 


Note: — There were no statistically significant differences between the treatment and comparison groups on any of the student 
characteristics. 
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Figure 4 shows the same information for fifth-graders. Again, the differences we observed on the 
student characteristics prior to propensity score analysis are largely gone, and the profiles of the 
treatment and comparison groups are almost identical. There are no statistically significant 
differences between the two groups on any of the six characteristics. 


4. Characteristics of treatment and control groups after propensity score analysis: 


5'" grade 
Group 
Treatment Comparison 
Student characteristics (N=331) (N=1,059) Difference® 
Gender Male 51% 50% 1 
Female 49% 50% -1 
Race/ethnicity American Indian 5% 5% 0 
Asian 8% 8% 0 
Black 41% 40% 1 
Hispanic 27% 27% 0 
White 20% 20% 0 
Eligible for free or reduced-price lunch 77% 78% -1 
English Language Learner 34% 34% 0 
Receives Special Education services 12% 11% 1 
MCA-III Math 2014 scale score: Mean 446 446 0 


@ Difference in percentage points (treatment minus control group). 


Note: — There were no statistically significant differences between the treatment and comparison groups on any of the student 
characteristics. 


This equalization between the groups makes it more likely that any treatment group difference 
that might be found in student achievement in spring 2015 could be a STEM Pathways effect. 


Achievement test results 


Analyses were conducted to test whether STEM Pathways had an effect on spring 2015 MCA-III 
results. A series of regression analyses were carried out to determine whether MCA math, reading, 
and science scores differed significantly between the treatment and comparison groups, using the 
IPTW weighting described earlier to neutralize any effects differences in the six student 
characteristics between the two groups might have on these scores. Findings from this analysis 
are reported in Figure 5, with fourth and fifth grade findings shown separately. For the most part, 
there is very little difference in MCA scores between the treatment and comparison groups. 
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Average (mean) scale scores are reported for each MCA test by group as well as the percentage 
that reached the proficiency level on the test (i.e., met or exceeded grade-level standards). 


5. Achievement tests (MCA-III) results in spring 2015: Treatment and comparison 


groups 
Group 
Treatment Comparison 
4" Grade (N=355) (N=1,067) 
Math Scale score mean? 446 448 
Percent proficient® 45% 50% 
Reading Scale score mean? ° 441 444 
Percent proficient®: ° 31% 37% 
Treatment Comparison 
5"" Grade (N=331) (N=1,059) 
Math Scale score mean? 543 543 
Percent proficient? 36% 37% 
Reading Scale score mean? 544 545 
Percent proficient? 40% 39% 
Science Scale score mean? 539 540 
Percent proficient? 28% 30% 


aln 4h grade, scale scores range from 409 through 499 in math and 411 through 490 in reading. 
6 Percent meeting or exceeding standards. 
¢ The difference between the treatment and comparison group is statistically significant (p<.05). 


4 In 5h grade, scale scores range from 515 through 586 in math, 517 through 591 in reading, and 501 through 599 in science. MCA 
science is only administered to fifth-grade students. 


In fourth grade, average scale scores and proficiency percentages are somewhat higher in the 
comparison group than the treatment group for both math and reading. These differences between 
groups on these measures are statistically significant for reading but not for math. In fifth grade, 
the average scale scores for math, reading, and science are identical or almost identical for the 
treatment and comparison groups. Proficiency percentages are also very similar between the two 
groups for all three tests. None of the differences between groups for fifth-graders is statistically 
significant. Note that half or fewer of the fourth-graders are proficient across subjects and study 
groups (ranging from 31% to 50%). Among fifth-graders, 40 percent or fewer are proficient 
across subjects and study groups (ranging from 28% to 40%). In science, 28 and 30 percent, 
respectively, are proficient in the treatment and comparison groups. 


Two additional propensity score analysis methods were used to examine potential treatment 
effects on the MCA tests. One was the Average Treatment Effect (ATE) method which matched 
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each treatment group member with up to three comparison group members using propensity 

scores based on the same six student characteristics. The other was a regression analysis method 
that used propensity scores based on the six characteristics as covariates, but without weighting. 
The results using these two methods were essentially the same as those using the IPTW method. 


Treatment vs. comparison group achievement results within student demographic 
categories 


Potential treatment-comparison group differences on the 2015 MCA tests were explored within 
student demographic categories using the IPTW regression method. These analyses were 
conducted separately within the following demographic categories: female, male, eligible for 
free/reduced-price lunch, ineligible for free/reduced-price lunch, ELL, non-ELL, Asian, black, 
Hispanic, and white. (There were too few students to conduct the analyses within the American 
Indian group). 


Results from these analyses are reported in Figures 6 and 7 for fourth- and fifth-graders, respectively, 
indicating average 2015 MCA-III scale scores in the treatment and comparison groups. Some 
statistically significant differences between the treatment and comparison groups emerged in 
several demographic categories. The strongest pattern is in the white subgroup. Average scale 
scores are significantly higher in the comparison group than the treatment group for all the MCA 
tests in both grades among white students. A similar but weaker pattern of differences in favor of 
the comparison group occurred for the subgroup ineligible for free/reduced-price lunch (i.e., 
higher-income subgroup). The only statistically significant differences for this subgroup are for 
fifth-grade reading and science. 


There are several other weak patterns within demographic subgroups in one grade. Differences in 
favor of the comparison group are quite small (3-4 scale points), but statistically significant. 
Among fourth-grade boys, average MCA scale scores in math and reading are significantly 
higher in the comparison group than in the treatment group. Similarly, among non-ELL fourth- 
grade students, average MCA scale scores in math and reading are significantly higher in the 
comparison group than in the treatment group. Finally, among fifth-grade Hispanic students, 
average MCA scale scores in reading and science are significantly higher in the comparison 
group than in the treatment group. 
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6. Achievement tests (MCA-III) results in spring 2015 for 4'"-graders within student 
characteristics categories: Scale scores 


MCA-III scale score 
mean? 
Treatment Comparison 
Student characteristics Test group group Difference® 
Gender Female (N=709) Math 448 448 0 
Reading 444 445 -1 
Male (N=713) Math* 445 449 -4 
Reading* 438 442 -4 
Free or Eligible (N=1,076) Math 442 443 -1 
reduced-price 
lunch Reading* 436 439 -3 
Ineligible (N=346) Math 461 464 -3 
Reading 455 458 -3 
ELL status ELL (N=477) Math 440 439 1 
Reading 434 434 0 
Non-ELL (N=945) Math* 450 453 -3 
Reading* 445 449 -4 
Race/ethnicity | Asian (N=97) Math 451 449 
Reading 447 443 
Black (N=522) Math 438 440 -2 
Reading* 434 438 -4 
Hispanic (N=422) Math 445 445 0 
Reading 439 439 
White (N=347) Math* 458 464 -6 
Reading* 452 458 -6 


ain 4h grade, scale scores range from 409 through 499 in math and 411 through 490 in reading. 
Treatment group score minus comparison group score. 


* The difference between the treatment and comparison group is statistically significant (p<.05). 
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7. Achievement tests (MCA-III) results in spring 2015 for 5'"-graders within student 
characteristics categories: Scale scores 


MCA-III scale score mean? 
Treatment Comparison 
Student characteristics Test group group Difference? 
Gender Female (N=691) Math 544 544 0 
Reading 546 547 -1 
Science 540 541 -1 
Male (N=699) Math 543 543 0 
Reading 542 544 -2 
Science 539 540 -1 
Free or Eligible (N=1,089) Math 540 539 1 
oe Reading 540 541 4 
Science 536 535 1 
Ineligible (N=301) Math 554 557 -3 
Reading* 556 561 -5 
Science* 551 559 -8 
ELL status ELL (N=471) Math 538 537 1 
Reading 535 536 -1 
Science 530 531 -1 
Non-ELL (N=919) Math 546 546 0 
Reading 549 550 -1 
Science 544 545 -1 
Race/ethnicity | Asian (N=108) Math 544 547 -3 
Reading 546 545 1 
Science 542 543 -1 
Black (N=561) Math 540 538 
Reading 542 540 
Science 537 534 
Hispanic (N=372) Math 541 541 
Reading* 539 542 -3 
Science* 534 538 -4 
White (N=275) Math* 552 557 5 
Reading* 555 562 -7 
Science* 551 560 -9 


ain Sth grade, scale scores range from 515 through 586 in math, 517 through 591 in reading, and 501 through 599 in science. 
5 Treatment group score minus comparison group score. 
* The difference between the treatment and comparison group is statistically significant (p<.05). 
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Note that the equalization of the treatment and comparison groups on the six student characteristics 
using weightings, as described earlier, was conducted with the full treatment and comparison 
groups being intact. When analyses are performed on subgroups within the two study groups 
(such as with white or non-ELL student subgroups) the equalization is compromised. For 
example, it cannot be assumed that the subgroup of white students in the treatment and comparison 
groups are equalized on the other demographic characteristics when analyses are carried out on 
this racial subgroup separately. 


To help address this issue, additional analyses were conducted on several student subgroups to 
statistically control for demographic differences between the two groups. This was accomplished 
using regression analysis and adding free/reduced-price lunch eligibility and sometimes ELL status 
as covariates plus interaction terms for each (i.e., the interaction of each with the study group 
variable), when appropriate and when sample sizes were large enough. These analyses were carried 
out for the following subgroups: white students (4 and 5" grades), Hispanic students (5" grade), 
boys (4" grade), and non-ELL students (4 grade). 


For the white subgroup, differences in MCA test results between the treatment and comparison 
groups are reduced when controlling for free/reduced-price lunch differences between the two 
groups, and are no longer statistically significant except for fifth-grade science. The interaction 
term in this analysis is not statistically significant. ELL status was not controlled because there 
were too few ELL students among whites. Results of this analysis presented in Figure 8 indicate 
the difference in average scale score between the treatment and comparison groups (average 
treatment group score minus average comparison group score). All of the differences shown for 
the white subgroup are negative indicating somewhat higher average scores in the comparison 
group than the treatment group. These results suggest that part of the differences in white 
students’ test results in favor of the comparison group that we saw in Figures 6 and 7 may be due 
to a lower proportion of white students eligible for free/reduced-price lunch in the comparison 
group than the treatment group. By controlling for eligibility for free/reduced-price lunch, the 
pattern of comparison group white students scoring higher on MCA tests than their counterparts 
in the treatment group is weakened substantially. 


Turning to the fifth-grade Hispanic subgroup, we controlled for differences in eligibility for 
free/reduced-price lunch and ELL status between the treatment and comparison groups. This 
results in changes in the average difference in MCA math, reading, and science scores between 
the treatment and comparison groups, but now none are statistically significant. For fourth-grade 
boys, differences between the treatment and comparison groups change slightly after controlling 
for free/reduced-price lunch eligibility and ELL status. The difference in math is no longer 
statistically significant, while the difference in reading remains statistically significant. For 
fourth-grade non-ELL students, differences between the treatment and comparison groups also 
change slightly after controlling for free/reduced-price lunch eligibility, and the differences for 
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math and reading remain statistically significant. The interaction terms in all of these analyses 
are not statistically significant. 


8. Results of additional analyses of treatment-comparison group MCA-III differences 
within selected student characteristics categories 


Difference in 2015 MCA-III scale scores between 

treatment and comparison groups (treatment 

Subgroup Grade Test group mean minus comparison group mean) 
White students 4 (N=347) Math -4 
Reading -3 
5 (N=275) Math -3 
Reading -5 
Science -7* 
Hispanic students 5 (N=372) Math 8 
Reading -6 
Science -2 
Male students 4 (N=718) Math -6 
Reading -5* 
Non-ELL students 4 (N=945) Math -5* 
Reading -4* 


* The difference between the treatment and comparison group is statistically significant (p<.05). 


In sum, no strong patterns of differences between the treatment and comparison groups emerged 
across grades and MCA tests within any of the demographic categories examined. Overall, 
differences in test results tended to be small and most often slightly in favor of the comparison 
group. A weak but consistent pattern of MCA score differences in favor of the comparison group 
was found among white students and students ineligible for free or reduced-price lunch (higher- 
income students). That is, differences in favor of the comparison group were consistent across 
grades and type of test in these two demographic categories, but the differences were generally 
small and most often not statistically significant. 
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